Build the Loss of Control Observatory end to end - #29

Establish the Python 3.12 package boundaries, locked uv environment, local quality gates, plain-language project guide, and honest initial operating documents required by the rest of the roadmap.

Constraint: The complete demonstration must fit a 7–8 hour implementation window.

Rejected: Separate services at the outset | deployment and coordination cost would delay the first end-to-end path.

Confidence: high

Scope-risk: narrow

Directive: Keep collection, classification, storage, reporting, and interface code behind the package boundaries introduced here.

Tested: make install; make check; uv build; git diff --cached --check.

Not-tested: No live API, database, or service behaviour exists in this foundation.

Related: #1

Load versioned non-secret YAML and ignored environment credentials through one validated settings object. Process variables override local files, missing names are actionable, and secret values remain redacted.

Constraint: Different commands need different credential subsets during incremental delivery.

Rejected: Read environment variables throughout the codebase | configuration errors would be late, inconsistent, and harder to test.

Confidence: high

Scope-risk: narrow

Directive: Load settings once at each command boundary and pass the validated object inward.

Tested: uv run pytest tests/test_config.py; make check; git diff --cached --check; .env ignore verification.

Not-tested: No deployment secret store or live provider credential was used.

Related: #2

Add a read-only, non-persistent access check with a narrow provider boundary and safe failure modes. Document local script-app setup and the rate-limit assumptions that later collection work must preserve.

Constraint: Reddit credentials remain local and are not available in this workspace
Rejected: Store the sample for inspection | access proof does not require retaining user content
Confidence: high
Scope-risk: narrow
Directive: Keep the access check read-only and non-persistent
Tested: make check (12 tests); uv build; missing-credential CLI path
Not-tested: Live authenticated Reddit request

Reddit approval cannot fit the demonstration window, so use Bluesky's documented public AppView for the working path while preserving Reddit as an approval-gated adapter. Record the rejected bypasses and validate the new response boundary before exposing a successful access result.

Constraint: Reddit Data API approval can take weeks
Rejected: Reddit .json endpoints | not an approved collection contract
Rejected: Browser scraping with residential proxies | bypasses access controls and is operationally fragile
Confidence: high
Scope-risk: moderate
Directive: Keep source-specific access behind adapters and do not enable Reddit without approval
Tested: make check (20 tests); uv build; live observatory bluesky check-access returned 10 posts
Not-tested: Bluesky rate-limit and outage behavior under sustained collection

Add a forward-only SQLite migration and repeatable CLI entry point before the live collector writes data. The schema keeps redacted source evidence immutable, constrains versioned scores, cascades derived records during erasure, and prevents audit-history rewrites.

Constraint: The local demo uses SQLite and must remain reproducible from an empty database
Rejected: Create tables ad hoc in repository code | hides schema history and weakens startup repeatability
Confidence: high
Scope-risk: moderate
Directive: Add schema changes through a new migration; do not edit an applied migration
Tested: make check (25 tests); migration CLI first and repeated run; uv build; wheel contains SQL migration
Not-tested: Concurrent migration attempts from multiple processes

Bluesky post URIs carry author identifiers, so collection now pseudonymises author and post identity before the repository boundary. Bounded pagination, retry policy, query isolation, immutable inserts, and counts-only CLI output make live runs safe to repeat and diagnose.

Constraint: Bluesky AT URIs and public post URLs contain author identifiers

Rejected: Store raw AT URIs for convenient retrieval | this defeats ingest-time pseudonymisation

Confidence: high

Scope-risk: moderate

Directive: Keep direct identifiers out of repository inputs, storage, logs, and command output

Tested: make check (33 tests); uv build; live collection repeated with 185 then 0 inserts

Not-tested: Live multi-page pagination because the demo configuration intentionally caps each query at one page

The Observatory now scores bounded batches with a packaged pilot-derived rubric and validates every provider result before storage. Prompt hashes, model identity, reasoning, token usage, configured cost, and timestamps preserve reproducibility, while malformed and failed items remain isolated in the dead-letter queue.

Constraint: Classifier input is minimised public post text processed under the provider workspace's active retention terms

Rejected: Accept loosely parsed model prose | invalid or out-of-range results could contaminate evidence scores

Confidence: high

Scope-risk: moderate

Directive: Treat scores as review priorities, not verified incidents; preserve strict output validation and prompt hashing

Tested: make check (40 tests); offline Anthropic HTTP contract; uv build with packaged prompt

Not-tested: Paid live Anthropic classification; image and transcript artifact inputs are not connected yet

Add high-recall prescreening, OpenAI structured scoring, persistent JSON run records, and a deterministic static report. The two-stage gate mirrors the CLTR method while keeping item failures isolated and all provider results reproducible.

Constraint: OpenAI credits are available; Anthropic is outside the demo scope

Rejected: Logit-weighted judging comparison | no label-free metric can establish accuracy

Confidence: high

Scope-risk: moderate

Directive: Keep prescreen and score eligibility tied to exact model and prompt versions

Tested: make check (48 tests); empty-database CLI report smoke; uv build

Not-tested: Live OpenAI request and non-empty live report

Document the two-stage CLTR-derived method, the OpenAI provider decision, privacy constraints, report operation, and why the logit-EV comparison is deferred without independent labels.

Constraint: External writing must be concise and understandable without engineering context

Rejected: Use self-consistency as an accuracy metric | it measures stability, not correctness

Confidence: high

Scope-risk: narrow

Tested: Documentation links and git diff whitespace checks

Not-tested: Independent methodological review

Interleave scheming and reaction signals while rotating across AI terms. This prevents the eight-query production bound from collapsing to one generic AI term and excluding reaction language.

Constraint: Live collection is capped at eight queries to limit load and runtime

Rejected: Increase the query cap | raises external load without fixing ordering bias

Confidence: high

Scope-risk: narrow

Tested: Bluesky collection suite (6 tests); focused ruff and strict mypy

Not-tested: Exhaustive relevance comparison across query schedules

Record the bounded live Bluesky and OpenAI run, including partial collection, idempotent retries, model costs, score distribution, and report output. Add a plain-language account of assumptions and the 10x and 100x operating changes.

Constraint: No source content, author identifiers, credentials, or model reasoning may enter Git

Rejected: Present model consistency as accuracy | no independent labels exist

Confidence: high

Scope-risk: narrow

Directive: Update validation figures only from a fresh documented run

Tested: Two live collections, three live prescreens, two live detailed-score runs, report generation, make check (48 tests)

Not-tested: Classifier accuracy, continuous scheduling, transcript authenticity, and incident deduplication

Use a normalized transcription of the full published Appendix C prompt, preserve the exact published prescreen labels and recall instruction inside an explicit local wrapper, and store every field in the Appendix C output schema.

Constraint: The report does not publish the complete prescreen prompt

Rejected: Describe both prompts as exact | only Appendix C is published in full

Confidence: high

Scope-risk: moderate

Directive: Do not alter published prompt files in place; add a new version and preserve hashes

Tested: make check (49 tests), including prompt provenance and full score persistence

Not-tested: Image-input parity because the collector remains text-only

Add an isolated deterministic benchmark that exercises ingestion, duplicate handling, derived records, static reporting, integrity checks, and durable aggregate metrics. A clean-database guard prevents synthetic fixtures from contaminating collected evidence or producing misleading rerun numbers.

Constraint: External provider throughput must not be inferred from an offline storage benchmark

Rejected: Send tens of thousands of live Bluesky or OpenAI requests | unsafe, costly, and confounded by third-party rate limits

Confidence: high

Scope-risk: moderate

Directive: Keep synthetic benchmark databases separate from collection databases

Tested: Ruff, mypy, and 52 pytest tests

Not-tested: Full 50,000-record run is recorded separately after this commit

Record the benchmark conditions, production boundaries exercised, operational measurements, artifact hashes, environment, and explicit exclusions. Link the evidence from the project overview so reviewers can distinguish local pipeline capacity from live provider performance.

Constraint: Generated data and HTML artifacts remain outside Git

Confidence: high

Scope-risk: narrow

Directive: Do not present this benchmark as Bluesky, OpenAI, accuracy, or deployed-service capacity

Tested: Independent row counts, artifact hashes, and SQLite integrity check

Not-tested: Concurrent writers, deployed hardware, backup recovery, or external provider limits

Add retry-bounded dead-letter replay with short SQLite leases, current prompt-and-model idempotency, aggregate lifecycle records, and an isolated offline failure cycle. The demo proves failure capture, recovery, and replay no-op behaviour without touching live credentials or adapters.

Constraint: Replay may incur provider cost and must remain bounded

Rejected: Reuse the ordinary pending selector | it would process healthy unattempted work instead of only failed items

Confidence: high

Scope-risk: moderate

Directive: Keep replay claims leased and never log post content or credentials

Tested: Ruff, mypy, 58 pytest tests, including concurrent SQLite claims

Not-tested: Process crash during an active live provider request

Persist a bounded historical volume check that separates insufficient history from a real low-volume warning and exposes an alert-friendly exit code. Add least-privilege CI with immutable action pins, locked dependencies, safe caching, and the full local quality gate.

Constraint: Volume changes are operational signals, not incident-prevalence estimates

Rejected: Alert on the first collection run | no baseline exists and the result would be noise

Confidence: high

Scope-risk: moderate

Directive: Investigate provider, query, and collector health before interpreting a volume warning

Tested: Ruff, mypy, YAML parse, and 62 pytest tests

Not-tested: Hosted GitHub Actions execution before push

Implement separate source and artifact retention windows, safe artifact-path validation, dependency-aware cleanup, and pseudonymous-author erasure. Document the data lifecycle, provisional legitimate-interest rationale, access limits, API processing, transfer review, and remaining deployment obligations in plain language.

Constraint: Filesystem and SQLite deletion cannot form one atomic transaction

Rejected: Cascade artifacts when source posts expire | it would collapse the distinct 90-day and 180-day retention windows

Confidence: high

Scope-risk: broad

Directive: Verify identity and resolve it to the existing HMAC before running erasure; include backups and recipients in deployed procedures

Tested: Ruff, mypy, YAML parse, and 65 pytest tests including cutoff and idempotency cases

Not-tested: Deployed backups, downstream recipients, access controls, or legal approval

Promote source and stage into durable run dimensions, emit safe stage-specific JSON metrics, preserve collection bounds and failure categories, and add a hard stored-post cap with request pacing. Add bounded OpenAI retry handling and keep failed items out of ordinary batches so recovery remains explicit.

Constraint: Live source and model providers impose independent rate and cost limits

Rejected: Increase config defaults globally | routine runs should remain small unless an operator explicitly raises bounds

Confidence: high

Scope-risk: moderate

Directive: Use one stable AUTHOR_HMAC_KEY for the full lifecycle of a live dataset

Tested: Ruff, mypy, and 67 pytest tests

Not-tested: Larger live Bluesky and OpenAI runs require local ignored credentials

Replicate CLTR's semantic and entity-assisted grouping as a versioned pipeline stage, while enforcing the configured date span across each complete group. Preserve memberships, representative decisions, manual-review routing, privacy invalidation, and operational metrics for later dashboard use.

Constraint: Candidate incidents must remain distinguishable from reports and from manually reviewed findings
Rejected: Reproduce pairwise-only date checks | CLTR documents transitive chains that merge distinct events
Rejected: Add scikit-learn and SciPy | the credible subset is small and the standard-library implementation avoids two large runtime dependencies
Confidence: high
Scope-risk: moderate
Reversibility: clean
Directive: Do not remove the complete-component date-span guard without re-evaluating the documented false-merge cases
Tested: make check (74 tests, Ruff formatting and lint, mypy)
Tested: empty incident-analysis CLI run with durable pipeline and analysis records
Not-tested: manual review decisions on the forthcoming live dataset
Related: #19

A query can return and store valid pages before a later page fails. Treat the run as failed only when every query errors and no page completed; otherwise retain an explicit partial outcome.

Constraint: A terminal status must not contradict durable successful writes
Rejected: Key failure only to posts_inserted | idempotent reruns can complete pages containing only duplicates
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Preserve the distinction between zero completed pages and partial query failures
Tested: make check (75 tests, Ruff formatting and lint, mypy)
Related: live collection run d0a9673d-d663-4ed0-a595-27295795242d

The scaled live run found one credential-shaped value in public post text. Redact high-confidence OpenAI, GitHub, and AWS credential forms before persistence, document the audited source correction, and record the completed 1,000-post pipeline metrics and public Bluesky pagination behavior.

Constraint: Public source text can contain live third-party secrets even when author identifiers are removed
Rejected: Keep the public value because it was not our key | the warehouse must not become a credential store
Rejected: Publish local run artifacts | they contain collected evidence and remain access-controlled under ignored paths
Confidence: high
Scope-risk: narrow
Reversibility: clean
Directive: Keep credential redaction before every persistence and logging boundary; extend patterns only with high-confidence forms to limit evidence loss
Tested: make check (76 tests, Ruff formatting and lint, mypy)
Tested: SQLite integrity and foreign-key checks on the final 1,255-post warehouse
Tested: boundary-aware credential scan across database, report, and 44 local log artifacts
Tested: final counts of 1,000 screenings, 39 detailed scores, zero DLQ items, and zero candidate incidents
Not-tested: Credential formats outside the three documented high-confidence families
Related: #19

The dynamic dashboard and external consumers need one authenticated, source-backed metric layer. Add versioned FastAPI contracts, separate liveness and readiness, privacy-minimised exports, saved views, safe operational events, and an isolated failure-recovery demonstration.

Constraint: Dashboard data and exports must never expose direct author identifiers or source URLs

Rejected: Query SQLite independently from browser code | metrics would drift and bypass access controls

Confidence: high

Scope-risk: moderate

Directive: Keep dashboard metrics and exports on the shared analytics repositories

Tested: pytest tests/test_api.py tests/test_warehouse.py tests/test_observability.py

Tested: ruff check and mypy on src and affected tests

Not-tested: Browser dashboard workflow; implemented in the next commit

Serve a dense two-view dashboard from the authenticated API so reviewers can explore trusted metrics while operators can inspect runs, privacy controls, schema health, safe logs, and recovery evidence.

Constraint: Collected text and operational data must remain behind one local service boundary\nRejected: Static charts only | would not support filtering, drill-down, export, or recovery inspection\nRejected: Separate frontend service | adds a trust boundary and build system without helping the local demonstration\nConfidence: high\nScope-risk: moderate\nDirective: Keep analytics, API responses, and exports on the shared repository queries\nTested: make check (89 tests), Chromium desktop/mobile acceptance, CSV export, saved view, DLQ recovery, SQLite integrity and foreign keys\nNot-tested: deployed TLS ingress, multi-user authorization, sustained concurrent load

Document how non-engineers use the evidence view, how operators prove recovery and data health, and which privacy and deployment controls remain outside the local demonstration.

Constraint: External documentation must lead with plain language and distinguish engineering controls from legal compliance\nRejected: Claim GDPR compliance | local safeguards do not establish a lawful basis, deployed access control, or backup erasure\nConfidence: high\nScope-risk: narrow\nDirective: Update validation evidence and deployment limits whenever the service boundary changes\nTested: Commands reconciled with CLI help; links and documented counts checked against the live warehouse\nNot-tested: Fresh-clone manual setup on a second machine

Add exact detection, impact, response, recovery, and verification steps for Bluesky rate limits and model-provider outages, while marking the real outage path as untested.

Constraint: Recovery commands must avoid printing source content or credentials\nRejected: Automatic unbounded replay | amplifies provider outages and permanent schema failures\nConfidence: high\nScope-risk: narrow\nDirective: Keep external-outage claims separate from the exercised local timeout path\nTested: Commands checked against current CLI and warehouse schema\nNot-tested: Real OpenAI outage and authenticated Bluesky rate-limit response

Clarify partial classification coverage and source-date semantics, repair Swagger under the strict CSP, and expose safe demonstrations for validation, recovery, and warehouse structure.

Constraint: Demonstrations must not send external requests or write synthetic evidence into the main warehouse
Rejected: Relax the CSP for FastAPI's inline Swagger initializer | weakens a demonstrated security boundary
Confidence: high
Scope-risk: moderate
Directive: Keep validation demonstrations on the production validator with fixed synthetic inputs
Tested: node syntax checks; ruff format and lint; mypy; 92 pytest tests; browser verification of Swagger, recovery, validation, and schema inspection
Not-tested: Cross-browser rendering outside the Codex Chromium runtime

Preserve the published-method score while adding a versioned competing-hypotheses pass, fixed review routing, recoverable storage, and an inspectable report-detail chart.

Constraint: Keep CLTR-style scores and historical Stage 4 counts comparable

Rejected: Replace the 0-9 score | removes the baseline needed for comparison

Confidence: high

Scope-risk: moderate

Directive: Treat both axes as model judgements, not calibrated probabilities

Tested: ruff format/check; mypy; node syntax; 97 pytest tests; five-report OpenAI run; SQLite integrity and foreign keys; Chromium detail-view verification

Not-tested: Accuracy against independent human labels

Related: #28

Add a plain-language, categorized inventory of the implemented collection, LLM, recovery, privacy, observability, authentication, scale, and change-control decisions.

Constraint: Do not present a validated local demonstration as a deployed production service

Confidence: high

Scope-risk: narrow

Tested: make check (97 tests, formatting, lint, and type checking)

Not-tested: GitHub Markdown rendering outside the standard README renderer